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ABSTRACT 

A study examined the assessment of instructional documents 
written by undergraduate engineering students in a 300-level technical 
writing course. The six documents were independently ranked by nine readers — 
three writing faculty, three engineering. faculty, and three technical writing 
professionals. Besides ranking the documents, the nine readers were also 
asked to designate levels of success, to write "in-text" comments and, 
finally, to provide post-text narratives, summarizing each paper’s strengths 
and weaknesses. The greatest consistency across all six papers emerged among 
the writing faculty. Except for Document 1, all of their rankings were either 
identical or within one level. For the engineering faculty, only Document 4 
was ranked within one level, and for the technical communicators, none of the 
six documents among the three readers was ranked the same or within one 
level. Of the three stakeholder communities, only writing faculty undergo 
long-term socialization and enculturation within their discourse community, 
and writing assessment is central to their professional day-to-day lives. In- 
text comments (n=1035) were coded and sorted into five major categories that 
emerged: content, structure, format, style, and graphics. For the post-text 
narratives, ■ the same five assessment categories emerged as central. Readers 
in all three groups placed a high value on accurate, accessible content 
appropriately targeted for the specified audience, along with a coherent, 
logical document structure, effective document design, an appropriate and 
clean style, and effective integration of graphic elements. Lists 5 
references. (NKA) 
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ASSESSING INSTRUCTIONAL DOCUMENTS: A COMPARISON OF WRITING 
FACULTY, ENGINEERING FACULTY AND WORKPLACE TECHNICAL 
COMMUNICATOR PRACTICES 



Introduction 

This presentation focuses on the assessment of instructional documents written by 
undergraduate engineering students in a 300-level technical writing course. The 
documents were independently ranked by nine readers — three writing faculty, three 
engineering faculty, and three technical writing professionals. Besides ranking the 
documents, the nine readers were also asked to designate levels of success, to write in- 
text comments and, finally, to provide post-text narratives, summarizing each paper’s 
strengths and weaknesses. The purpose here was to explore the assessment perspectives 
of three stakeholder communities that intersect and overlap, and yet have differing 
professional and disciplinary cultures. I was interested to see whether consensus would 
emerge both within groups and between groups. 

I must say that I began this study with certain expectations. I thought that I would 
certainly see consistent patterns of assessment, depending on the disciplinary community 
of the particular readers. Instead, I was surprised to see a high level of inconsistency 
among the engineering faculty and the workplace professionals in regard to ranking. 
However, a more consistent pattern emerged within these groups and, indeed, among all 
nine readers in the success level designations as well as in the in-text and post-text 
comments. 
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Assessment Theory 

Before further discussing the study, I want to touch briefly on the theoretical 
underpinnings driving writing assessment today. Historically, writing assessment has not 
experienced as rich a development as research in composition theory and pedagogy. 
Clearly, however, the move from indirect to direct assessment has been pervasive. 
Traditional writing assessment practices, which often centered on indirect assessment, 
were based on a positivist epistemology that assumed a reality, in the words of Cuba, 
“driven by immutable natural laws” (Cuba, 1990, p. 19). The assumption was that student 
writing ability is a fixed, consistent, and acontextual trait. 

With research by Freedman (1984), White (1985), Williamson and Huot (1993), 
Huot (1996), and others, assessment theory has shifted to reflect our current 
understanding of writing as a constructivist, situated act. Specifically, the shift is to 
assessment procedures that are 1) site based; 2) locally controlled; 3) context sensitive, 4) 
rhetorically based, and 5) accessible (Huot, 1996). The approach in this study is grounded 
in such an understanding of assessment. 

Method 

The Papers 

Six papers were selected based on similar document length, 5 to 7 pages each, and 
similar subject matter — software or hardware procedural documentation. Specifically, the 
six papers focused on the following procedures: 1) building a motor control circuit for a 
robot, 2) installing al hard drive, 3) using an oscilloscope; 4) using Visual C++, 5) 
making an animated movie using Flash software, and 6) upgrading RAM on a PC. 
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But why instructional documents? To begin with, as a genre instructional 
documents are central to the field of technical communication, and, consequently, are 
important to all three stakeholder groups that assessed them. In the work environment 
outside the academy, evaluation focuses on the performance of complex and meaningful 
tasks that contribute to the well-being of the organization and to the larger community. 
This was precisely the aim of the instructional documents written by engineering 
students — to aid target readers to perfonn meaningful tasks related to real-world contexts. 

The Audience 

As to the audience and purpose of the documents, the assignment for the student 
writers read as follows: “Assume that your readers are college peers and that they need to 
learn the procedure covered in your instructions, but they know almost nothing about the 
device and its operation or the procedure they must learn to perfonn.” 

Instructions for Readers 

This, then, was the purpose and the audience for the students’ writing task. Now 
Fd like to shift my discussion to the method underlying the assessment of the six 
instructional documents. I began by securing three writing faculty, three engineering 
faculty, and three workplace technical communicators to assess the papers. 

I did not provide a specific rubric to the readers because that would necessarily 
impose my set of criteria and values on the papers. As Huot (1993) notes, a 
predetennined rubric increases the chance of consensus among readers, but this 
consensus can come at a price of less authentic assessment. Instead, I asked the readers to 
consider the papers from their own professional perspectives. 




4 



Paper draft CCCC 02 
4/24/2003 2:51 PM 



5 



The Readers 

A few more words about the readers: Two of the three writing faculty have 
master’s degrees and one has a Ph.D., each with over 15 years of teaching experience. All 
three engineering faculty have Ph.D.s, each with over 20 years’ of teaching experience. 
Of the three workplace professionals, one is a technical writer with a bachelor’s degree 
and 8 years’ experience, the second is a usability specialist with a master’s degree and 6 
years’ experience; and the third is a documentation manager with a master’s degree and 
18 years’ experience. Two of the writing faculty are female as is one of the technical 
professionals. All three engineering faculty are male. 



Findings 

I’d now like to speak to my findings — that is, 1) the ranking outcomes, 2) the 
level of success designations, 3) the extent and nature of in-text comments, and 4) the 
nature of the post-text narratives made by the nine readers. 



Readers’ Ranking of the six papers 

The greatest consistency across all six papers emerged among the writing faculty. 
Except for Document #1, all of their rankings were either identical or within one level. 
For the engineering faculty, only Document #4 was ranked within one level, and for the 
technical communicators, none of the six documents among the three readers was ranked 
the same or within one level. 

Consistency of Ranking of Six Instructional Documents within Groups 



Rank 


Writing Faculty 
Document Choices 




Engineering Faculty 
Document Choices 




Technical Comm 
Document Choices 


1 


Upgrad 


Upgrade 


Build 




Build 


Build 


Make 




Use 


Build 


Build 




e RAM 


RAM 


Robot 




Robot 


Robot 


Flash 




oscillis. 


Robot 


Robot 




Doc 6 


Doc 6 


Doc 1 




Doc 1 


Doc 1 


Movie 




Doc 3 


Doc 1 


Doc 1 
















doc 5 
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Make 
Flash 
Movie 
doc 5 


Build 
Robot 
Doc 1 


Upgrade 
RAM 
doc 6 




Upgrade 
RAM 
Doc 6 


Upgrade 
RAM 
Doc 6 


Use 
oscillis 
Doc 3 




Upgrade 
RAM 
doc 6 


Run 
C/C++ 
doc 4 


Run C7C++ 
doc 4 
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Use 


Make 


Make 




Run 


Make 


Install 




Build 


Use 


Make Flash 




oscillis. 


Flash 


Flash 




C/C++ 


Flash 


hard 




Robot 


oscillis. 


M ovie 




Doc 3 


Movie 


Movie 




doc 4 


iVI ovie 


drive doc 




Doc 1 


Doc 3 


Doc 5 






doc 5 


doc 5 






doc 5 
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Run 


Run 




Install 


Upgrade 


Use oscillis. 




Robot 


oscillis. 


oscillis. 




oscillis. 
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C/C++ 




hard 


RAM 


Doc 3 
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Doc 3 


Doc 3 




Doc 3 


doc 4 


doc 4 




drive 
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Reasons for ranking variability 

What might be the reasons for the ranking variability among the engineering 

faculty and the technical communicators? One might assume that members of the same 
discourse community would rank papers similarly, that their training and growth in their 
particular professions would result in a similar ranking order of instructional documents. 
However, only the writing faculty had a community of colleagues with whom they 
regularly interacted in regard to evaluative measures, scoring, and writing assessment 
issues. The three groups clearly had different levels of experience with writing 
assessment. Writing faculty undergo long-temi socialization and enculturation within 
their discourse community, and writing assessment is central to their professional day-to- 
day lives. For engineering faculty and technical communicators in the workplace, while 
they too are stakeholders in what defines a good instructional document, they do not 
discuss assessment issues among themselves in the direct and central way that writing 
faculty do. Furthermore, the papers had not been pre-selected based on writing quality 
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levels. That is, the ranking was made doubly difficult because the quality range was not 
as wide as it might have been with a pre-selected group of very weak papers and very 
strong papers. 

Readers’ Assessment of the Documents as highly successful, 
successful, marginally successful, or unsuccessful 

This is evidenced by the way readers responded to the second task, designating 
each document as highly successful, successful, marginally successful, or unsuccessful. 
These terms correspond roughly to the traditional grading paradigm scheme of A, B, C 
and D, where A is equal to a 4.0, B to a 3.0, and so on. [OVERHEAD] 

Readers’ Rating of Documents According to Levels of Success 





Writing Faculty 


Engineering Faculty 


Technical 

Communicators 


# of papers 
judged highly 
successful 


4 


2 


2 


# of papers 

judged 

successful 


6 


11 


9 


# of papers 
judged 
marginally 
successful 


7 


5 


6 


# of papers 

judged 

unsuccessful 


1 


^0- 


1 


Level of 
success 

translated into 
a 4.0=A scale 


2.72 


2.83 


2.66 



In this framework, the highest average score for the six papers came from the tln'ee 
engineering faculty (2.83), and the lowest came from the teclmical communicators (2.66), 
with the writing faculty in the middle (2.72), all, however, in the B-/C+ range. 
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Ultimately, the distinctions between groups as to the level of success of the papers were 
minimal. 



Readers’ In-Text Comments 

In addition to ranking the documents and assigning levels of success to them, the 
nine readers also provided in-text comments on each of the papers. I first completed a 
count of all the comments made — some 1035, or an average of 23 comments per paper, 
ranging from surface error notations to extended queries and suggestions. What was 
interesting is that only 25 of these comments included praise or positive feedback, while 
1010 comprised questions, corrections, and concerns. I then coded the comments, sorting 
them into five major categories that emerged: Content, structure, format, style, and 
graphics. 



NATURE OF NARRATIVE ASSESSMENT COMMENTS 



Praise Comments 

Questions/Concems 

Total 



25 Comments 
1010 Comments 
1035 Comments 



1. Content (content questions; unclear , incomplete or incorrect info 



2. Structure(sequencing, order of document parts) 



3. Format/Layout/Design 

Non-parallel terms for headings, numbering of lists, lack of sufficient 
lists, bullets vs. numbers; general layout, spacing 
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4. Style 

• Usage/dictioii/telegraphic prose, Inconsistent nomenclature 

• Mechanics (punctuation, capitals, spelling) 

• Syntax/predication/redundancies, lack of sentence-level clarity 

• Sentence boundaries 



5. Graphics 

• Unclear, weak graphics 

• Inconsistent labeling of graphics/unlabeled graphic elements, 
unnamed figures 

• Missing graphics 



Content included any reader comment that asked a question about the subject matter of 
the document or pointed out unclear procedural steps. It also included comments about 
incomplete or missing infonnation, and general procedural accuracy. The structure 
category included comments on logical sequencing; proper placement of cautions and 
warnings, and orderly progression of document parts. The format category included 
comments about layout and document design — for example, sufficient use of lists, and 
good use of white space. The fourth category, style, encompassed any comment on usage, 
mechanics; syntax; or sentence boundary issues. And the final area that emerged was the 
effective use of graphics to facilitate clarity in the instructions. 

What was interesting was the distribution of in-text comments among the tliree 
groups of readers. [TRANSPARENCY] 

Percent of comments in each of six categories 





WF (360 in-text 
comments) 


EG (285 in-text 
comments) 


TC (390 in-text 
comments) 


Praise % 


3% 


1% 


1% 


Content % 


24% 


16% 


40% 


Structure % 


4% 


3% 


4% 
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Format % 


15% 


6% 


9% 


Style % 


46% 


71% 


36% 


Graphics % 


8% 


3% 


10% 


Total % 


100% 


100% 


100% 



Writing faculty and technical communicators made approximately the same number of 
comments on the six documents (360 and 390 comments respectively), and their 
distributions were similar, although the writing faculty made fewer remarks on content 
and more on format and style. The engineering faculty made fewer in-text comments as a 
group — a total of 285, and of those comments, a surprisingly high 71% reflected a 
concern with elements of style. Perhaps this group had fewer comments on content since 
they were familiar with the content and thus had fewer questions about this central 
element of the documents. But while the distribution varied between groups, all nine 
readers included in-text comments on the five assessment elements that emerged — 
content, structure, format, style, and graphics. 

Post-Text Narratives across All Three Reader Groups 

This was further supported by the readers’ post-text nairatives on each paper. 
Here, as with the in-text comments, the same five assessment criteria emerged as central. 
First, all nine readers valued accessible, purposeful content delivered at an appropriate 
level for a specific, target audience. The readers judged papers as weak and ranked them 
low when the stipulated audience did not match the level of the instructions. Second, all 
nine readers valued a coherent structure — a clear introduction, with a statement of 
purpose, a preview of major steps, and lucid sequencing of infomiation. Third, all nine 
readers admired a good fomiat with subdivided sections and procedures and effective use 
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of white space. In regard to visually dense, unpacked infomiation, one reader wrote, “My 
eyes ache trying to chunk the information.” Fourth, all nine readers valued a clear, readable 
prose style and objected to first draft quality text. In response to one such document, one 
reader commented, “Looks like a hastily prepared, coiTupted Word file. Some parts are so 
badly written that I know they weren’t proofread.” And fifth, all nine readers alluded to 
the visual impact of each document and applauded the liberal use of illustrations and 
graphics. 

Conclusion 

In summary, while the ranking of the six documents was inconsistent among two 
of the three groups, the level of success designations, the in-text comments, and the post- 
text naiTatives suggest that the nine readers did use similar criteria in assessing the 
instructional documents: In addition, while Huot (1996) found that writing faculty tend to 
ignore the communicative elements of writing and focus on its stmctural aspects, I did 
not find this to be home out in this study. Indeed, for all three groups, the structural 
aspects played a role, but nonetheless a subordinate role to the “communicative aspects,” 
that is, the accessibility and relevance of the message for the audience, the content itself 
Readers in all three groups placed a high value on accurate, accessible content 
appropriately targeted for the specified audience, along with a coherent, logical document 
structure; effective document design; an appropriate and clean style, and effective 
integration of graphic elements.. The differences that did emerge suggest that these three 
stakeholder communities that intersect and overlap in their professional interests — 
writing faculty, engineering faculty, and technical communicators in the workplace — can 
continue to learn from each other about the assessment of instructional documents. 
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